fix(oasis): implement prompt-level tool calling in the CLI bridge (agents could never act) - #29
Open
jcartagenamartinez-legaltech wants to merge 2 commits into
Conversation
The CLI model bridge dropped tool schemas entirely
("CLIModel ignores tool schemas") and always returned plain-text
completions with finish_reason=stop. But OASIS agents act on the
platform exclusively through tool calls (perform_action_by_llm
consumes response.info['tool_calls']), so with CLI providers agents
could never execute a single platform action: simulation rounds
stayed silently empty (observed: 2-4 active rounds out of 15 across
every run), and the few recorded actions came from env-scripted
seeds. Interviewed agents even reported it verbatim: "the social
platform tools were not loaded".
Since CLI providers cannot do native function calling, implement it
at the prompt level:
- Serialize the tool schemas (name, description, JSON parameters)
into an instruction appended to the conversation, asking for a
single JSON object: {"tool_calls":[{name, arguments}]} to act, or
{"content": "..."} for a plain reply.
- Parse the CLI's text response (tolerating markdown fences, JSON
embedded in prose, stringified arguments and OpenAI-style nested
{"function": {...}} entries) and rebuild an OpenAI-shaped
ChatCompletion with proper message.tool_calls entries
(id, type=function, function.name, function.arguments as a JSON
string) and finish_reason="tool_calls" — exactly the contract
CAMEL's ChatAgent consumes to execute FunctionTools.
- Normalize tool-protocol messages on the way in (role=tool results
and assistant tool_call echoes become plain text) so multi-turn
tool loops survive the round-trip through a text-only provider.
- Graceful degradation: a response with no parseable tool call
falls back to plain content, preserving old behavior for
interview prompts that explicitly forbid tool use.
Tested: 7-case parser suite (fences, prose-embedded JSON, string
arguments, nested format, content fallback), completion-builder
assertions against the exact fields CAMEL reads
(function.name / json.loads(function.arguments) / id), and message
normalization for tool-role and tool-call-echo messages.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
- Replace the greedy {.*} regex fallback with a raw_decode scan from
each '{': finds the first VALID JSON object even when the model
wraps it in prose containing braces (the greedy match spanned from
first to last brace and lost valid tool calls).
- Filter parsed tool names against the offered tool list: CAMEL
indexes self._internal_tools[name] without a guard, so a
hallucinated name would raise KeyError and lose the agent's whole
round. Unknown names are dropped with a warning instead.
- Trigger the tool-call echo summary on the PRESENCE of tool_calls
rather than content is None: CAMEL serializes assistant tool-call
messages with content='' (never None), so the branch was dead and
multi-turn context was silently lost.
- When the model attempts the protocol but yields neither valid
calls nor string content, return empty content instead of leaking
the raw JSON blob as agent-visible text.
Re-ran the parser/builder/normalizer test suite against the updated
module: 7/7 parser cases plus the CAMEL-contract assertions pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Author
|
Empirical validation — before/after on the same real scenario We re-ran the exact market-simulation scenario (189 personas, Twitter + Reddit, 16 rounds,
Action histogram after the fix: We also ran the stratified interview path from #28 on top of this branch: 50/50 interview transcripts completed, zero empty responses. Happy to share redacted logs if useful. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
The CLI model bridge drops tool schemas entirely (
CLIModel ignores tool schemas) and always returns plain-text completions withfinish_reason=stop. But OASIS agents act on the platform exclusively through tool calls —perform_action_by_llmconsumesresponse.info['tool_calls']— so with CLI providers (claude-cli/codex-cli) agents can never execute a single platform action.Observable symptoms (every run we executed, 119-185 agents): only 2-4 active rounds out of 15, with the recorded actions coming from env-scripted seeds rather than agent decisions. When we interviewed agents post-run they reported it verbatim: "the social platform tools were not loaded — no like, no comment".
The fix
CLI providers can't do native function calling, so it is implemented at the prompt level, entirely inside
oasis_llm.py:{"tool_calls":[{"name", "arguments"}]}to act, or{"content": "..."}for a plain reply.arguments, and OpenAI-style nested{"function": {...}}— and rebuilt as an OpenAI-shapedChatCompletionwith propermessage.tool_calls(id,type=function,function.name,function.argumentsas a JSON string) andfinish_reason="tool_calls": exactly the contract CAMEL'sChatAgentreads (chat_agent.pybuildsToolCallRequestfrom those three fields).role=toolresults and assistant tool-call echoes become plain text) so multi-turn tool loops survive a text-only provider.Testing
contentfallback, garbage-text fallback.function.name,json.loads(function.arguments),id,finish_reason).Relationship to other PRs
Independent of #26/#27/#28 in diff terms (only touches
app/utils/oasis_llm.py). On Windows, CLI providers also need #26 to run at all.🤖 Generated with Claude Code